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(57) Abstract 

In a data communication system 
having network traffic with flows, a net- 
work trafhc director system using router 
modules. The router motlules direct inif- 
fic based on optimizing a merit function or 
penalty function to reduce costs of conties- 
lion for stochastically changing demands 
and flows in the data communication sys- 
tem. The router modules exchange values 
with neighlK)ring router modules. Based 
on the exchanged values and values lo- 
cal to a router module, flow conditions 
are checked and if necessary the local val- 
ues are adjusted until the flow conditions 
are satisfied or a time period expires. Ad- 
justments are associated wilh optimizing a 
merit function or penally function. Based 
on the adjusted values, the router mod- 
ule adjusts parameters to be used to di- 
rect packets of the network traffic tlows 
to other router modules or other destina- 
tions within the data communication sys- 
tem. An aggregation scheme is used for 
reducing the number of values stored in a 
single router module. 
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METHOD AND APPARA TUS FOR NETWORK CONTROL 

TECHNICAL FIELD 

The present invention is related generally to data communication 
systems and, in particular, to network control of data communication systems. 

5 BACKGROUND Ol THE INVENTION 

Data communication systems generally consist of multiple networks that 
are linked together with bridges, switches, routers, or other devices. These linking 
devices direct data packets along various paths from an origination to a destination. 
Since a data communicalion .system tends to be large with many networks and 

1 0 interconnecting links, typically a data packet can lake many paths from its origination lo 
its destination. A goal of a network traffic director system is lo direct the Journey of 
each data packet from its origination to its destination so that the capacity of the 

netwiirk is eii'iciently utilized. 

I raditionul network traffic director systems utilize a plurality of network 
15 routers based on Internet Protocol (IP). The IP routing approaches are based on a 
Shortest Path (SP) method in which packets are .sent lo their destinations along the 
shortest paths in the network. There are various methods lo determine shortest paths 
among the IP routing approaches. However, these approaches all share a common 
assumption that if each packet travels along the shortest path, then the overall workload 
20 required for the routing will remain as small as possible. 

The assumption of traditional routing systems would hold true if the 
overall performance of a routing system could be adequately measured solely on the 
basis of the lolal number of decisions that routers need to perform. However, other 
factors are involved in the overall performance of a routing system. One of these 
25 factors is the amount of congestion in a data communicalion system. Some theoretical 
proposals exist which attempt lo address the congestion of network traffic director 
systems. These approaches, however, are impractical in providing adequate solutions to 
the congestion problem. The end result of some of these approaches produces, on 
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average, flows that are little different than the shortest path methods of IP routing 
taking congestion into account. Another theoretical approach to congestion is intended 
lor environments having predefined aspects, such as loading and nctv^ork topology. 
However, a data communication system is seldom sufficiently predictable for these 
5 predefined approaches. 

Other theoretical approaches attempt to address an aspect of the 
stochastic (random) nature of flows in a data communication system by allowing for 
randomness in the arrival times of individual data packets into a linking device. These 
approaches, however, are still formulated in a deterministic manner by requiring that 

10 demand or load rales and flow rales in other parts of the communication system be 
known in advance oi their actual occurrence. 

These prior art delerministic approaches are not ideally suited for data 
comnuiniealion system environments in which demands or loads and flows in other 
portions of ihe data communication system have stochastic distributions due to 

15 changing demands or loads of individual users or due to any multiplexing involved. 
This means that not only are the arrival times of individual packets random, but also the 
rales of the demands or loads and flows in other portions of the data communication 
system are random and not known before their actual occurrence. 

An efl'eciive optimization method must not only minimize congestion in 

20 the data communication system but it must be robust enough to adapt to sudden changes 
in conditions of the data communication .system such as packet arrival, loading on the 
system, and system topology without predefined scenarios of when or how these 
changes will occur. 

In data communication systems, loading changes quickly and 

25 dramatically due to such influences as changing demands or loads of individual users 
and multiplexing methods such as time division multiplexing (TDM) where time slots 
of system use are divided among individual users. With TDN4, a data communication 
system constantly changes the particular users on the system at any given moment, thus 
loading changes rapidly. Network topology can also have rapid stochastic changes due 

30 to situations such as equipment failures or weather conditions. 
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Also, whereas the prior art systems and methods focus on a single 
indicator, such as shortest path or average delay, as a basis to optimize data flow, there 
are many additional factors. These include average queue length at various data 
communication system linking devices, variance or standard deviation in individual 
5 delays of data packets traveling from an origination to a destination, and average and 
variance oJ' the individual utilized capacity of links and linking devices in a data 
communicuiion sy.stem. Thus, a system and method is greatly desired which optimizes 
packet How in a data communication system based on numerous factors while being 
robust enough to effectively adapt to unexpected events. 

10 SUMMARY Of 1 Hli INVHN TION 

I he present invention is directed to a network traffic director .system in a 
dat;i communicaiion system having network traffic with randomly distributed demands 
aiul Hows in one a.spect of tlic present invention the network traffic director system 
includes router modules ci)iifigured to direct data packets in the data communication 

\> svstem. each router module being a home router module including a neighborhood 
^,nper^•i.s<»r I he neighborhood supervi.sor is configured to send home potentials of the 
iiome n>uier module to neighborhood supervisors of neighboring router modules and to 
receive neighbor potentials of llie neighboring router modules from neighborhood 
supervisors of neighboring router modules. The home router module further includes a 

20 dynamic load balancer configured to determine Hows based on the home and neighbor 
potentials. The dynamic load balancer adjusts the home potentials if first conditions 
including flow conditions are not met. The dynamic l(»ad balancer also updates routing 
tables if second conditions based on the adjusted home potentials are met. The home 
router module also includes a dynamic data How splitter configured to receive data 

25 packets from networks and router modules. The dynamic data flow splitter selects a 
portion of the data communication system for each data packet received based on the 
updated routing tables. Each received data packet is transmitted to the portion of the 
data communication system selected by the dynamic data flow splitter for the received 
data packet. 
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Another aspect of the invention includes the home potentials of each 
homo router module being associated with a pair of nodes of the data communication 
system including an origination node and a destination node. The home potentials of 
each router module arc also associated with a quality of service level. The 
5 neiehborhood supervisor is configured to send the home potentials to neighborhood 
supervisors of neighboring router modules when the flow conditions are not met. The 
How conditions include conservation of flows. The dynamic load balancer is 
configured to determine Hows by using a response function associated with optimizing 
at least one of a penalty function or a merit function for stochastic demands or fiows of 

10 the data communication syslem. The dynamic data fiow splitter is configured to select 
ihc portion of the data communication system for each packet received further based on 
cither a Markov protocol, a Routing Wheel protocol, previously selected portions of the 
data communication when an address for a received data packet is a same type as an 
address stored in the dynamic data How splitter, or on a quality of service level 

I 5 ass(Kiated with a data packet. 

in a further aspect of the invention the neighborhood potentials are 
steered according to an aggregation scheme. The merit or penalty functions are at least 
locally approximated by a quadratic function of the Hows. The ideal data fiows are 
further based on resistance of arcs as.sociated with the router module wherein each 

20 resistance of an arc is configured to be a function of either arc capacity or a function of 
fiows of the arc. The neighbor router modules are configured to be associated with the 
home director module through at least one of a political, organizational, topological, 
topical, or geographical relation. In a further aspect each router module is configured to 
adjust home potentials to an equilibrium point for a difference function associated with 

25 fiows of arcs associated with the router module. 1'he difference function is associated 
with differences between home potentials and neighbor potentials. The merit function 
or penalty function involves costs of congestion including at least one of traffic delays, 
high latency, diminished throughput, lost packets or unresponsiveness to sudden 
changes in topology or loading. 
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BRIEF DFSCRIP riON OF THE DRAWINGS 

Figure I is a topology diagram representative of a network traffic 
director system of the present invention. 

Figure 2 is a block diagram showing interconnection detail of router 
5 modules of the network traffic director system of Figure 1 . 

Figure 3 is a block diagram showing internal detail of a router module of 

Figure 2. 

Figure 4 is a topology diagram showing internal detail of a router 
module of Figure 2. 

]() Figure 5 is a relational diagram of an aggregation scheme used in the 

network traffic director system of Figure 1 . 

Figure 6 is a topology diagram showing internal detail of an arc o< a 

router module of iMgure 2. 

Figure 7 is a fiowchart of a method used by the router modules of 

15 l*igurc2- 

i)i-;taifi-:i) description of i he invention 

Large communication data systems are comprised oi* a colleclion oi 
networks linked together by linking devices, such as routers, bridges, and switches 
known herein as router modules or director modules. The depicted embodiment in 

20 Figure 1 has a network traffic director system 110 in a data communicaiion system 
which links a colleclion of networks 120 via router modules 130 and associated links 
140 and links 150. A link 140 is a data communication path for data packets to travel 
between a network 120 and a router module 130. A link 150 is a data communicaiion 
path for data packets to travel between two router modules 130. Network traffic is 

25 made up oi' individual data packets which are sent between the networks 120 via the 
router modules 130. The network traffic director system 110 as embodied in the router 
module 130 allows more efficient utilization of data communication systems over prior 
art linking devices. The router module 130 is configured to adapt to sudden changes in 
the data communication network, such as randomly and rapidly changing service 
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demand or load rales or topology, while optimizing utilization of the data 
communication system based on numerous factors, including traffic delays, queue 
lengths, variance in individual delays, average and variance of utilized capacity of links 
and linking devices, number of lost packets, latencies, and throughput of linking 
5 devices. 

Unlike prior art methods, for ihe depicted embodiment, the optimization 
methods used account for the numerous factors influencing utilization of a data 
communication system by formulating a penalty function (which accounts for negative 
effects, such as lost packets, increased delays, high latency, diminished throughput, 

10 amonu other things) or its opposite merit function (which accounts for beneficial 
elTecls). The merit and penalty functions are formulated for optimization and 
determination of equilibrium points given a rapidly changing, stochastic environment of 
random demands upon a data communication system, random flows through a data 
c(Mnmunication system and random changes in topology of a data communication 

15 syslem. 

This stochastic approach is far different from prior art methods, which 
rely on predictable, non-stochastic environments and utilize deterministic formulations. 
The systems and methods used by the depicted embodiment of the network traffic 
director system 110 are designed to be effectively implemented in a data 

20 communication system without having extraordinary processing requirements, resource 
needs or other demands upon the overall data communication system, l^iesc and other 
advantages will become apparent in the following detailed description. In the following 
description, numerous specific details are provided such as methods and systems used 
to collect information from the router modules 130, to determine whether conditions are 

25 satisfied, and to adjust values related to the router modules 130 accordingly. One 
skilled in the relevant art, however, will recognize that the invention can be practiced 
without one or more of the specific details, or with other devices and methods. In other 
instances- well known structures or operations are not shown or described in detail to 
avoid obscuring the description of the embodiments. 
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In order to connect the networks 120 of the data communication system 
togclhi-T- tlic router modules 130 are connected together via communication units 210 as 
shown in Figure 2. The communication units 210 are linked together by data 
communication pathways 212. The communication units 210 in one embodiment arc 

5 l-:thcrnet network cards and the data communication pathways 212 are Ethernet cables, 
luhernei is a term well known in the art referring to collision sense multiple 
;icccs.s/collisic)n detection techniques as described under the IEEE 802 series 
specifications. In other embodiments of the present invention, the communication units 
210 anil pathways 212 are other types of point-to-point data communication links. 

](, Some of the communication units 210 shown in Figure 2 are not shown 

to he connected via any data communication pathway 212. These communication units 
21(1 can K- connected io other router modules 130 which are not shown in Figure 2. 
I he luiinKT of communication units 210 shown for each individual router module 130 
is noi inieiuleil lo limit the depicted embodiment since some router modules have more 

1.^ coiniiiunication units 210 than shown in Figure 2 if necessary to link additional router 
nu>dules. 

l-ach router module 130a-d. as shown in Figure 2, is connected to a data 
network 1 2()a-d. \ lowever. the number of data networks 1 20 connected to any one 
router module 130 varies, and .some, router modules 130 are only used to interconnect 
20 other n>uler modules and are not directly connected to any network 120. In other 
instances, router modules 130 can be directly connected to a plurality of networks 120. 
Each router module 130 has a neighborhood supervisor unit 214 that includes a values 
e.\changer. a dynamic load balancer unit 216, a routing table unit 218 with a plurality of 
routinu tables, und a dynamic data How splitter unit 220. 

25 The neighborhood supervisor unit 214 of a home router module 130 

exchanges information with router modules that are neighbors to the home router 
module. These neighboring router modules 130 are directly connected to the home 
router module. For instance, in Figure 2 the neighbors of home router module 130a 
include router modules 130b, 130c, and 130d because they are directly connected to 

30 router module 130a via communication pathways 212 from one communication unit 
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210 of a neighboring router module 130b-d to a communication unit 210 of the home 
router module 130a. 

The neighborhood supervisor unit 214 also monitors for malfunctions 
and link failures in the data communication system. Along with monitoring, the 
5 neighborhood supervisor unit 214 updates data in the dynamic load balancer 216 of the 
home router module 130 to be consistent with the current physical stale of the data 
communication system. The neighborhood supervisor unit 214 also reports detected 
network lai lures to a central fault reporting device located in the data communication 
system tlirough a central fault monitoring function. 

10 The dynamic load balancer unit 216 uses information collected by the 

neighborhood supervisor unit 214 of the home router module 130 from the neighboring 
router modules 130. The dynamic load balancer unit 216 adjusts the routing tables of 
the routing table unit 218 based upon the information collected in order to optimize 
overall utilization of the data communication system served by the network traffic 

15 director s\ stem 1 10. The methods u.sed by the dynamic load balancer unit 216 will be 
discussed furlher below. The routing table unit 218 provides information to the 
dynamic data flow splitter unit 220 of the home router module 130 in order that the data 
packets received by the communication unit 210 of the home router module 130 are 
directed to destination networks 120 or router modules 130 in a manner that preserves 

20 the integrity of the mathematical models used by the dynamic load balancer unit 216. 

The dynamic data flow splitter unit 220 performs the actual routing 
operation of data packets received by the home router module 130 to direct the received 
data packets to other networks 120 or router modules. The division ol' labor between 
the routing table unit 218 and the dynamic flow splitter unit 220 can be varied. In one 

25 embodiment, the routing table unit 218 prepares the routing data in a suitable form for 
the dynamic data How splitter unit 220. The routing table unit 218 then stores this 
routing data until the dynamic data flow splitter unit 220 requires the data. The 
dynamic data flow splitter unit 220 then looks up the routing data in the routing table 
unit 218 to determine how a particular data packet is to be routed, 'fhc dynamic data 

30 flow splitter unit 220 in various embodiments, of the present invention use different 
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methods lo determine how a received data packet should be routed. Depending on 
which method is used, a received data packet will be sent by either the communication 
unit 210 or the dynamic data flow splitter 220 of the router module 130 to different 
router modules 130 through one of the communication units 210 or to a directly linked 
5 network 120. Four different methods of determination, also called splitting protocols, 
are typically chosen. 

Markovian splitting is the simplest of these routing protocols to 
implement in software. In the Markovian protocol, the routing table unit 218 
transforms the desired Hows into equivalent routing fractions. Only the flows out of the 

10 router module 130 are relevant. For each destination, if / is the desired How to the 
neighbor / for all the neighbors / of the router module 130, then the fraction of the How 
lo this neighbor is P,-f,/^ iJ\ ^^^^rn, the dynamic data How splitter unit 220 

views the fraction as a probability of a Markov process. When routing a data packet, 
the dynamic data flow splitter unit 220 looks up the probabilities corresponding to the 

1 5 packet destination. />]. P2 Pk- The flow splitter then "casts a random die/' with an 

associated probability p, of the outcome /, and sends the data packet to the lucky 
winner. 

The easiest way to cast the random die is lo generate a uniformly random 
number between /.ero and one and determine where this number iails in the cumulative 

20 probability sequence { I\] given by /V; - 0. = P,,\ + The random numbers can be 
generated "on the fly*' or they can be stored a priori. 

Tlie second routing protocol, called a "Routing Wheel/' reduces the 
randomness t)f the Markovian protocol and is easy lo implement in hardware. Given 
the desired probabilities p/, /7i. . . . Pk-. for flows to 1 through K neighboring router 

25 modules 130 of a given router module, the routing table unit 218 of the given router 
module generates N (c.^.. 100) labels. The labels identify individual router modules 
130 and are assigned among the neighboring router modules 130 according to the 
probability associated with each neighboring router module. For instance, the 
neighboring router module 130 having probability /?/ is assigned Np, number of 

30 labels identifying that particular neighboring router module. The routing table unit 218 
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then arranges the labels into a conceptual distribution resembling a physical circle 
where all label types are distributed fairly evenly around the conceptual circle. The 
roulinu table unit 218 then sets a conceptual pointer to a location on the circle. When 
the dynamic data flow splitter unit 218 services a data packet, it looks up the currently 

5 selected label on the circle. The dynamic data flow splitter unit 220 then sends the 
packet to the neighboring router module 130 indicated by the current label and then 
advances the pointer to the next label on the circle. The process never runs out of labels 
since when ihe pointer is at the label in position M the pointer next advances to the 
label in position 1. In llirther embodiments, a routing wheel is generated for each 

10 originalion/desiinaiion pair and quality of service level combination (OD/OoS). 
( )1) (JitS i:ou\b\\vduous are discussed in detail below. 

Some higher level IP protocols are designed in such a way that makes 
the arrival oi the data packets in an orderly sequential fashion at the final destination 
desirable. I he.se protocols can handle fallouts and out-of-sequcnce arrivals, but they 

15 become increasingly inefficient as the number of out-of-sequence arrivals growls. In 
order {o increase ihe efficiency of such transfer protocols, the following modification of 
flow spliliing is used. This modification is incorporated into the third routing protocol 
and works with either Markovian splitting or flouting Wheel splitting. 

1 or the third routing protocol, the granularity of splitting is changed for 

20 the applications running the higher level protocols mentioned above. Such protocols 
can usually be recognized from a device number in a packet header. The dynamic flow 
splitter unit 220 is endowed with short-term memory. When one of the "suspect^" 
device numbers appears in the header of a packet, the dynamic How splitter unit 220 
remembers the origin, the destination, the device number of this packet, as well as the 

25 label of the neighboring router module 130 to which the packet was sent. All 
subsequent packets that have the same origin, destination, and device number that arrive 
while the memory persists are then sent to the same neighboring router module 130 as 
the original packet. In alternative embodiments, tlie short-term memory is optionally 
refreshed by the later arrivals. 
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The roiirth routing protocol used with the dynamic data flow splitter unit 
220 relates to low latency and quality of service. Latency of a system is the time 
required to send and receive information. Some applications, such as telephony, video- 
telephony, or real-time interactive games or simulations require low latency levels. The 
5 latency requirements of other applications such as transmissions of email are not as 
stringent- The ordinary method for lowering latency for urgent data packets is to give 
these packets a routing priority while in routing queues as discussed below. 

Achieving additional speedups of urgent data packets is realized by 
making the dynamic data flow splitter unit 220 send more of the urgent packets over 

10 faster links and paths. The dynamic data flow splitter imit 220 then compensates for 
this by sending more of the ordinary packets over slower links. I'he compensation can 
be adjusted to leave the overall performance of the routing system unchanged. In one 
embodiment, idenlificalion of the f aster links by the network traffic director system 1 10 
in the data communication system is readily accomplished because a high correlation 

15 exists between the highly used links and the faster links, 'fhis eondilion is used to 
satisfy latency and quality of serviee requirements by directing the urgent packets 
toward the most frequented path choices for a given address group and by directing 
more of the less urgent packets away from the most frequented path choices. 

The router module 130 also has a quality of service unit 310. a loll gate 

20 unit 312. an encoder-decoder unit 314, and a central fault monitoring unit 316 as shown 
in Figure 3- 1 he quality of service unit 310 is concerned primarily with latency 
regarding transmission of data packets in the data communication system. The quality 
of service unit 310 is used to establish separate queues for applications with different 
latency and bandwidth needs. The queues are then served by appropriate frequencies of 

25 service and processing rales. Also, billing rales are dependent upon the level of quality 
of service, fhe toll gate unit 312 collects the billing statistics used to bill customers of 
ihe data communication syslem. These statistics primarily focus on billing. However, 
ihey can be used to deteiTiiine bandwidth demands and optimize routing network 
topology. 
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To ensure privacy of communication, the data packets sent across 
network links that arc vulnerable to eavesdropping are encoded for transmission and 
decoded at the destination router module. The encoder-decoder unit 314 performs the 
encoding and decoding operations. This link security is not directly related to the 
5 routing operations of the router module 130. Sufficient bandwidth for the encoder- 
decoder unit 314 is required so thai transmission speed of the overall router module 130 
is not impeded. 

As described above, the dynamic load balancer unil 216 uses 
information from the neighborhood supervisor unit 214 to determine parameters that the 
]{) routing table unil 218 then uses to prepare routing table data. A goal of the dynamic 
load balancer unil 216 is to oplimize traffic Hows in the data communication system of 
which the network IralTic director system 1 10 is part. A method used by the dynamic 
load balancer 216 ol'lhe depicted embodiment used to accomplish this goal is described 
as follows. 

15 As discussed above, each link 140 between one router module 130 and 

one network 120 and each link 150 between two router modules 130 is a data 
communication path that allows for two-way traffic of data packets. Thus, each of links 
140 and 150 of I 'igure 1 allows data packet traffic to travel in two directions either to or 
from a particular network 120 or router module 130. 

20 To assist in describing the systems and methods below, each direction 

that a packet can travel along a link 140 or 150 is referred to as an "arc.'^ 'fhe term 
"arc** does not refer to any particular physical part of a link 140 or 150 because the links 
allow packets to travel in either one of two directions. Rather, the term ^^arc^^ is 
conceptual since it refers to only one direction of travel in a link 140 or 150. Since 

25 links 140 and 150 each allow for two directions that packets can travel, each link 140 
and 150 is assigned two conceptual arcs, one arc for each of the two directions of travel 
allowed for within each link. I hus, each line of Figure 1 which represents a link 140 or 
150 could be replaced by two lines representing two arcs which conceptually indicate 
the two directions of travel that are possible for packets to take on the link. The arcs of 
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links 140 and 150 arc illustrated in Figure 4 where each link has two arrows 
rcprescniing arcs for the two directions that packets can travel for any one link. 

The depicted embodiment is not limited to links between networks 120 
and router modules 130 but could include links between other network devices or 
5 networks of any size and topology. For example, links between routers and individual 
computers or terminals and links between large networks such as the world-wide 
Inlernci and a country sized corporate networks and small office local area networks are 
included in the depicted embodiment. 

Within router modules 130 there is an In node 420 and an Oui node 430 

1 0 as sh()W!i in Figure 4. Flow of the internal arcs 440 originate from the In nodes 420 and 
terminate at the Oia nodes 430 of each router module 130. The internal arcs 440 allow-' 
(or additional parameters to he used in optimizing the data communication sy.stem. To 
furnish additional parameters, each In node 420 and Out node 430 is treated similarly as 
how the router modules 130 are treated in the various formulations for the depicted 

I 5 emhodimenls discus.sed below. 

lor typical flow conditions in a data communication system, an overall 
How in a particular are typically is a conglomeration of one or more separate Hows. 
Facli separate llinv is characterized by its particular combination of How origination 
and ilow destination expressed as its origination/destination (OD) pair, fhe origination 

20 and destination are typically expressed at the network 120 or router module 130 level, 
so il is typically not expressed at a level specifying particular devices attached to a 
network. In some embodiments, the separate Hows are characterized only by OD pairs, 
so Ibr these embodiments, the separate ilovvs for a particular arc have unique OD pairs. 
Summing over these unique OD pairs gives the overall flow for these arcs. 

25 In the depicted embodiment, the separate flows are also characterized by 

different quality of service {OoS) levels. In this embodiment, at least occasionally, a 
particular arc wilt have more than one separate Ilow having the same OD pair, but 
having different OoS levels. To determine the overall flow for these types of arcs, 
the separate flows would be summed over both the OD pairs and QoS levels, giving 

30 unique OD/OoS combinations. Characterization of the separate flows of an arc is not 
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limited in the present invention to only OD/QoS combinations, but also incorporates 
other factors associated with traffic How in a data communication system. For instance, 
in another embodiment, characterization of separate flows in an arc includes a device 
number of an origination or destination device attached to a network 120. 
5 The overall (low of an arc is also constrained by an upper bound and a 

lower bound. The upper bound is typically proportional to the processing capacity of 
the arc for the depicted embodiment {c.g.^ 0.7 or 0.8 of the processing capacity of the 
arc). Hach individual (low can also be constrained by its individual upper bound and 
individual lower bound. Again, the most common lower bound is zero for the depicted 
1 0 embodiment. 

The method lo optimize traffic llow used by the dynamic load balancer 
unit 216 of the depicted embodiment defmes an ideal theoretical slate for data packet 
traffic in tiie data communication system as the case where there is zero flow 
throughout the data communication system while all demands or loads by users upon 

\5 the data communication system are satisfied. A solution for data Hows in the data 
commuuicalion system is sought where the solution is a point in a solution space that 
has a minimum distance to the theoretical ideal stale. The solution is subjecl lo certain 
physical constraints of a data communication system and is found through convex 
duality theory or convex oplimizxition theory. 

20 In the depicted embodiment, the solution for data flows in a data 

communication system is uniquely formulated lo optimize a variety of network 
objectives and conditions compared with prior art formulations that focus on one known 
data rale ob)cclive or condition subject to conservation of data flows. Data flows are 
conserved when the amount of flow coming into a node (typically a router module 130) 

25 equals the amount of flow leaving the node. Any data either produced or destroyed at a 
node is factored into data flow conservation. 

The solution for data flow^s also optimizes the following uniquely 
formulated expression E^^ ^{f) involving a substantially quadratic function of data 
flows in a data communication system: 



30 
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where the subscript indicalcs a particular OD/OoS combinalion and the subscript 
'-^//)" indicates a particular arc '^ah'\ The term fj^ah is for flow associated with a 
5 particular OD/OoS combination and a particular arc "'ab^\ The terms ij^,,^ and ^j^ah 
are resistance and goal parameters, respectively, for the particular ("^^D/Oo.S" combination 
and arc '\fh'\ The multiplication factors a and \\ are used to tailor emphasis 
between the two component expressions for Since only the relative 

magnitude of u and \\ are important, we implicitly assume the quantities are scaled to 
10 salislV (X - 1. 

In the form above, the expression Z:'^^ ,^{/) is a penalty function which 
represents undesirable inlluences and results affecting communication How in a data 
communication system as have been described. Optimization of the expression 
^ penalty function would result in minimization of {/). If. the 

15 expression Z*-,^ (./ ) wiis negated it would take on the form of the merit function. 
Optimization of a merit function would result in a maximi/,alion of the merit function. 
Other formulations that are equivalent to optimization of the penalty and merit 
functions involve determining a point of equilibrium. As known in the art. once an 
optimization formulation is described, an equivalent formulation for a point of 
20 equilibrium is determined by using mathematical techniques known in the art. 

In a preferred embodiment, /y = /•„/, = \/cah', }^/\ah .^ah = 0: and a and p 
are of the same magnitude. Then the solution of the optimization problem is the same 
as the solution of the competitive equilibrium problem using only the latter part of the 
equation, that is 
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over ihc non-supply/demand arcs. 

Other embodiments also include minimization of other penalty functions 
and maximization of other merit functions. Such penalty and merit functions are related 
to the depicted embodiment in that portions of these other penalty and merit functions 
5 are ai least locally approximated by the expression (/) above. These penalty and 
merit functions all are advancements compared with the prior art, since they are used to 
optimize Hows in a stochastic environment of a data communication system. 

The expression lAf) incorporates factors associated with individual 
OD/OoS combinations for each arc ^'ah^' over all the arcs in a data communication 
10 system. Hach indi\'idual OD/OoS combination '^f of an arc ''^///' has a particular 
resistance parameter, related to any hindrances to How and a goal parameter ^j^^j/^ 
related to the current demand or load by users. 

For the depicted embodiment, the resistance parameter for an individual 
arc "nh" and OD/OoS combination is also unique!) formulated, being proportional 
15 to the reciprocal of the overall data How capacity, f of the individual arc 'V//)" and 
OD'XJoS combination "/". Thus, oc \ IC ^^^f,. where C).,,/, is the maximum amount of 

data How which the individual arc ^^ah" is capable of supporting for OD/QoS 
combination '^r - For instance, Cah = (^).„f, and f\j/, ^ l/C/^^^/,lbr all /. In 

component tlows. most commonly <?^'^'~^c^/-^'''. The resistance parameter is also used 

7 

20 io designate that an arc for an OD/OoS combination is not functional by setting f),,,/, to a 

relatively large number. The solution to the optimization of the uniquely formulated 
expression E^^ ^ ( / ) over all the component flows//,^;/, of each OD/OoS combination '7" 

for each arc "r//)'' over all arcs and OD/OoS combinations of a data communication 
system results in solutions of flow for each OD/OoS combination "7*" for each arc 

25 "ah" in the data communication system. 

Methods that could be used to solve the optimization of the uniquely 
formulated expression include methods that have been used to solve other 

optimization formulations such as a projected gradient method, and a hill climbing 
method. The depicted embodiment uses a unique distributed optimization method 

30 incorporated in the dynamic load balancer unit 216 for optimizing the expression 
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^ui\{/) This mclhod is uniquely formulated for stochastic network flow that is 
uncertain, rapidly changing and random and topology thai can stochastically change. 
The distributed optimization mclhod is generally easier to implement and converges 
more quickly to a solution compared with prior art methods that have been used on 
5 other Ibrmulations. The distributed optimization method also incorporates conservation 
of flows and unique formulations of^j.^ih ^"d /y,^/,. 

As part of this distributed optimization method, the depicted 
embodiment defines a set of quantities al each router module 130 that are related to 
ilcnvs. Each quantity is referred to as a ^ poieniial." The set of potentials for a 

10 particular router module 130 has one potential for each possible OD/OoS combination 
known to the particular router module 130. Possible OD pairs are determined by the 
various combinations of a data packet origination and a data packet destination. These 
cc^mbinations wt>uld not typically include combinations where both the origination and 
destination are the same. I ypically. the data packet origination and destinations would 

IS be networks 120 and router modules 130, but could include other network devices al a 
more discreic le\el as discussed above, such as individual computers or terminals. 
Thus, ihe po.ssible OD pairs of the r^D/a^.S' combinations for a particular router module 
130 would typically include the OD pairs derived from all data packet originations and 
data packet destinations known to the particular router module 130. 

20 Typically networks 120 and router nu:tdules 130 are at limes data packet 

originations and al other times data packet destinations. As such, the same two 
components of a flrst OD pair could be the components of a second OD pair by 
designating the flrst pair origination as the second pair destination and the flrst pair 
destination as the second pair origination. For small sized data communication systems. 

25 a router module 130 stores one potential for each possible (;Z)/Oo*V combination for the 
entire data communication system. For a larger sized data communication system, an 
aggregation scheme is used which reduces the number of potentials stored by each 
router module 130. 

An aggregation scheme reduces the number of networks 120 and router 

30 modules 130 known to any one router module 130 in order to keep the set of potentials 
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stored in the router module to a manageable size. As shown in an aggregation hierarchy 
5 1 0 of Figure 5, this aggregation scheme has a physical portion 5 1 2 with physical nodes 
and a conceptual portion 514 with conceptual nodes. The physical nodes include 
networks 1 20 and router modules 1 30. The conceptual nodes each serve as labels for a 
5 particular collection made up of any number of physical nodes and/or other conceptual 
nodes. The physical nodes of the physical portion 512 are typically found in a 
particular geographical region. Figure 5 only illustrates a representative example, and 
the depicted embodiment can have other regions of various sizes and locations. 

Although Figure 5 illustrates an aggregation hierarchy in terms of 

10 geography, the depicted embodiment is not limited to only hierarchies defined in 
Licographical terms. Other hierarchies defined in terms of an organizational, political, 
topical, topological, or other types of structures are afso included in the present 
invention. Also, Figure 5 is not intended lo limit the present invention to any particular 
manner or type of geography in dcfming the physical portion 512 of the aggregation 

15 hierarchy 510. Tiie piiysical nodes can be grouped according to other common 
geographical regions besides the manner of using cities as regions as illustrated in 
Figure 5. 

For illustration purp(xses. Figure 5 shows the regions of Seattle. Portland, 
San Francisco (S.F,), Fos Angeles (F.A.). Boston, New York (N.Y.), Atlanta, and 

20 Miami having physical nodes 520-534 respectively. The Seattle region and Portland 
region represented by conceptual nodes 540 and 542 are further aggregated into the 
Northwest (N.W.) region represented by conceptual node 560. Similarly, conceptual 
nodes 544 and 546 for the S.F. and L.A. regions respectively are further aggregated into 
the .southwest (S.W.) region represented by conceptual node 562. Conceptual nodes 

25 548 and 550 for the Boston and N.Y. regions respectively are aggregated into the 
Northeast (N.F.) region represented by conceptual node 564. Conceptual nodes 552 
and 554 for the Atlanta and Miami regions respectively are aggregated into the 
Southeast (S.E.) region represented by conceptual node 566. The conceptual nodes 560 
and 562 for the N.W. and S.W. regions respectively are aggregated into the West Coast 

30 region represented by conceptual node 570. The conceptual nodes 564 and 566 for the 
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N.n. and S.K. regions respectively are aggregated into the East Coast region represented 
by conceptual node 572. The conceptual nodes 570 and 572 for the West Coast and 
East Coast regions respectively are aggregated into the USA region represented by the 
conceptual node 574, 

5 IMiysical nodes of the various regions in the physical portion 512 are 

directly connected with other physical nodes (typically router modules 130) in other 
regions of the piiysical portion. For instance, a physical node 520c in the Seattle region 
is directly connected to a physical node 522a in the Portland region through link 580. 
l lic \ arious regions are connected together via this type of direct connection between 

10 physical nodes in one region and other physical nodes (typically router modules 130) in 
nnt>ihcr region. Not all physical nodes in one region need to be connected to another 
ph\Nical tUK\c in anollier region. 1 his is because physical nodes within the same region 
arc ct)nnccied to one another either directly or through other physical nodes (typically 
ri)uier nH>dules 130 1 in the same region. Subnetworks formed by the aggregation 

\f> .scheme need maintain connectivity. Thus, between each pair of nodes in a 
subnet uork there must be a connection. In the aggregation scheme illustrated in Figure 
5. the physical nodes 520a-c in the area oT Seattle all share a common conceptual node 
540 in the conceptual portion 514 labeled Seattle. Each physical node 520a-c in the 
area of Seattle stores one potential for each possible OD/OoS combination based on the 

20 physical nodes 520 composed of networks 120 and router modules 130 in physical area 
512 assDciaied with the Seattle conceptual node 540. Also, under the same aggregation 
scheme, each physical node 520a-e in the Seattle area further stores one potential for 
each possible OD/OnS combination based also on the conceptual nodes that are 
blackened in on I igure 5. The resultant OD/OoS combinations include any OD pair 

25 combination based on either physical nodes, conceptual nodes or a mi.Kture of both 
physical and conceptual nodes as the originations and destinations of the OD pairs. The 
conceptual nodes that are blackened in on Figure 5 are Seattle conceptual node 540, 
rV^rtland conceptual node 542, N.W. conceptual node 560. S.W. conceptual node 564, 
West Coast conceptual node 570, East Coast conceptual node 572, and USA conceptual 

30 node 574. As shown in Figure 5. the physical node 520c is directly connected to 
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physical node 522a via physical link 580 and physical node 524a via physical link 582, 
Due lo these direct conncclions, physical node 520c also stores potentials for OD/OoS 
combinations that incorporate one or both of physical nodes 522a and 524a as cither 
origination, destination or both origination and destination. Thus, physical node 520c 
5 stores more potentials due to more OD pairs than the other physical nodes 520. In an 
alternate embodiment only physical nodes in the same region as the node in question 
along with appropriate conceptual nodes arc used for possible OF) pair combinations. 
Tor instance in the alternative embodiment, physical node 520c only stores potentials 
for physical nodes 520a-c in physical portion 512 and the blackened conceptual nodes. 

10 In the akernative embodiment, physical node 520c does not store potentials for any 
oihcr tlircctly connected nodes such as physical nodes 522a and 524a. 

There are general rules lo designate which physical and conceptual 
nodes arc used Id derive OD pairs for the OD/OoS combinations for a particular 
physical node, first, a router module 130 (at physical node 520c in the example of 

15 Figure 5) stores potentials for each OD/OoS combination having an OD pair based on 
the physical nodes (520b and 520c in the example of Figure 5) with which the router 
module l.'^O shares a common conceptual node (540 in the example of Figure 5). 
Second, il" the router module (at physical node 520c in the example of Figure 5) is 
directly connected to other physical nodes (522a and 524a in the example of Figure 5). 

20 the other physical nodes are also used to derive other OD pairs for OD/OoS 
combinations ibr which potentials are stored in the router module (at physical node 
520c). Finally, the router module (at physical node 520c) will store potentials for 
()D/0()S combinations for OD pairs that incorporate one or both of conceptual nodes 
thai are either "lineal ancestors'* of the router module (at physical node 520c) or 

25 ^'siblings" of the lineal ancestors. For the example of Figure 5, the conceptual nodes 
that are lineal ancestors of the physical nodes 520 (including router modules 130) are 
Seattle conceptual node 540, N.W. conceptual node 560. West Coast conceptual node 
570 and USA conceptual node 574. The conceptual nodes in the example of Figure 5 
that are siblings of the lineal ancestors are Portland conceptual node 542 (sibling of 

30 Seattle conceptual node 540), S.W. conceptual node 562 (sibling of N.W. conceptual 
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node 560). and East Coast conceptual node 572 (sibling of West Coast conceptual node 
570). 

The method for optimizing network How in a data communication 
system used by the depicted embodiment of the dynamic load balancer unit 216 is 
5 further based on a difference function d(fi,ahPj,ah)' 1 difference function <:/(//',;/>.P/Ui/>) 
is evaluated for each arc ^'ah'^ for each OD/OuS combination for each router module 
130 in the data communication system. When the difference function difjahPi.ah) »s 
evaluated for a particular router module 130. that router module is designated a home 
router module. The difference function dUjMhPjMh) '^^ evaluated for each arc that either 

10 oriuinaies from or terminates with the home router module. Router modules 130 that 
are physicallv connected to the home router module are designated neighboring router 
modules. I he neighboring router modules share with the home router module 130 the 
ares either originating or terminating with the home router module potentials. 
Foienlials p^,, and are at the origination and destination router modules 130 

1 5 respcetivelv of arc ah !or OlVOaS combination *:/"". fhus. the home router module 1 30 
and neighboring router modules 130 are either origination or destination router 
modules, depending upon the origination and destination of the particular arcs being 
evakiated. 

Kor a particular arc 'Uth" involved in the evaluation, the difference 
20 function cKf,,ahP,Mh) evaluated for each common OD/QoS combination "/"^ that is 
known both to the home router module 130 and the neighboring router module 130 of 
tlie particular arc ^\ihr The resultant expression for the difference function c/{ff ah,pi^ah) 
lor arc ^V//>" and C^D/CAaV combination is 



25 



^'(fjMh . P/Mh ) = (''/.ah / 2)* (k,,/, - / ) - //.ah * (PjM - PjJ> ) 



where 

{ 
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The potentials pJ^^ and pjh are potentials for router modules 130 'V and "^h"' 
respectively ibr the 'y^^ OD/OoS combination. Flows ./i,,,/^ where k^J are flows for the 
arc ^V/A" associated with 0/:>/6>(aS' combinations other than C^D/C'fAV combination y\ 

The first term, {r,^„, /l)* {}:^^, - f^ ^,,J , of the difference function 
5 <^[j],,H^Pfoh ) ^ simplified version of the expression for Ea.{\{f,.ah) where only one arc 
"a/r and one OD/OoS combination y is considered. The difference function 
il^ff .th'l^f.iih'^ is subject to the following boundary conditions: 



10 where 



and where and /,;/, are lower limits, and and //„/,are upper limits on and 

1 5 rcspcciivcly. 

An intuitive explanation oi'the difference function diff oh^pjab) is that the 
first term related to Ku.\\{f/.uh) represents undesirable results and conditions of data 
packet traffic being transmitted over a data communication system, such as delays, poor 
uiili/ation of resources, poor adaptability to sudden changes, etc. For a given arc, when 
20 tiie goal parameter, jC',,/.^ equals /.ero and the resistance parameter equals the inverse of 
capacity of the arc, then the first term of the difference function c/(Jj^ahPj^ah) equals the 
number of jobs per unit lime that have to go into a waiting queue because a router 
module 130 is busy. The second term - p/./,) represents effective utilization of 



wo 00/24164 PCT/US99/24145 

23 



the data communication system as subject to constraints imposed by the data 
communication system such as flow constraints. The difference function J(Jj,ahPj.ab) 
thus represents a difference between desirable and undesirable aspects involved with 
the daia communication system. 

5 As pan of finding a local equilibrium point, by differentiating the 

difference function (l(f,,„i,.Pj.ah) al each arc between router modules with respect to the 
How /i„,, of the OD/OoS combination ';/" and arc ''cib'\ setting the resullant differential 
equal lo zero, and solving for the fiow, /,•„/.. a response function rcsult.s for flow 
As ilkisiraled in Figure 6. for a particular arc -cth'^ 610 having beginning router module 

1 0 "<i" 620 and ending router module "h" 630 and for a common OD/OoS combination "f\ 
tiie respon.se function of llie optimal flow „/, 640 is: 



n 
If 



wilcrc min and max are minimiim and ma.ximum functions which clioo.sc the minimum 
i)r maximum term oi" the pair of terms that are biackelcd. For instance, max («,/') 

15 chooses the maximunt value between a and h. Thus, max (a. min (h.c)) first cltooses 
the minimum between h and c and then chooses the maximum between a and the 
minimum choice between h and c. Minimum How for /,.„/, is typically zero. Maximum 
How Ibr /,..,/. is typically a percentage of arc capacity. 

The goal parameter. .C'„/,- '""xJ resistance parameter, /v.,,/,. are for the 

20 particular arc "ah" and OD/OoS combination "/'■ The potential p,,,. is for the arc 
beginning router module "a" 620, and the potential /y/, is for the arc ending router 
module "h" 630. Both potentials /7,.„ and p,j, are associated with a common OD/OoS 
combination ':/". Typically, the origination and destination nodes for an OD/QoS 
combination arc not the arc beginning and arc ending router modules 620 and 630 

2-^ respectively. Also, the origination and destination nodes for an OD/OoS combination 
can be a combination of physical and/or conceptual nodes whereas the arc beginning 
and arc ending nodes are always physical since they are router modules 130. 

Summing these optimal flows for arc beginning "a" and arc ending '"h" 
router modules 620 and 630 for arc over the (9D/C)aV combinations that are known 
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to the arc beginning and ending router modules results in the optimal flow for arc '"ah". 
This response function of How describes an optimized flow based on certain values for 
potentials. These potential values further satisfy a constraint based on how data packets 
are created and terminated in the data communication system. 

5 In the depicted embodiment, the potentials are independent variables and 

the flows arc dependent variables. However, in other embodiments, different subsets of 
(lows and potentials arc used so that some flows and potentials are dependent variables 
and other ilovvs and potentials are independent variables. 

1 he depicted embodiment of the invention is configured so that all data 

10 packet traffic originates and terminates at network terminals of network 120 or router 
modules 130. This means that all data packet traffic having a common destination, 
either created at a particular router module 130 or arriving at the particular router 
module, must ec|ual the data packet traffic wMth the common destination leaving the 
router module 1 30. 1 his requirement is referred to as a conservation of flows or a flow 

IS constraint. When implementing the method of the depicted embodiment, the router 
modules 130 scattered throughout a data communication system frequently adjust the 
data packet routes in the data communication system to pursue the goal of maximizing 
net benefits for stochastic distributions of network loads and stochastic changes in 
topology. I he adjustments are done by manipulating potential values according to the 

20 conservation of Hows. 

Adjustments are first needed when a data communication system 1 10 is 
initially brought into service. Adjustments are subsequently needed due to the 
continually changing demands or loads by users of the data communication system and 
changes in the topology of the data communication system due to growth of the data 

25 communication system or equipment failures within the existing data communication 
system, fhe method of the depicted embodiment incorporated into the dynamic load 
balancer units 216 of the router modules 130 is responsible for these adjustments made 
by the router modules. 

As stated above, the dynamic load balancer unit 216 uses information 

30 collected by the neighborhood supervisor unit 214. This collected information can now 
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be described in terms of the response function for flow, as discussed above. The 
discussion will be focused on a particular router module 130 designated the "home" 
router module, however, the home rouler module discussed is representative of all 
router modules in that all router modules also perform the steps described. 
5 In general, there are two main approaches that are used in the present 

embodiment to determine the Hows for a given home router module 130. In both 
approaches, arcs are distinguished as being either supply arcs or non-supply arcs. A 
supply arc citlicr directly connects an origination network 120 or origination user device 
of a network for a given ()D/OoS combination to the given home rouier module 130 or 

10 a router nK)dule between the network or user device and the home router mode where 
there is no branching of the arcs for the home router module and the network or user 
device. N(^n-supply arcs are also directly connected to the given home rouler module 
but are not directly connected to an origination node. 

l or both approaches, the goal parameters for a supply arc for a given 

1 5 ()l)/0(tS combination are set equal ip jhe user demand or load placed on the supply arc. 
l-'or instance, if user demand ox load was 100 Mbits/sec on a supply arc for a given 
OD/OaS combination then the goal parameter for the same supply arc for the same 
r;/;/(;o.V combination would be set to 100 Mbits/sec. The neighborhood supervisor unit 
214 senses what line demand or load is from any networks 120 on network devices that 

20 arc directly cimnected to the home rouler module 130. Also, for both approaches the 
resistance parameter for a given ^>>D/C^^aV combination would be set to a relatively large 
number lor a supply arc. The large number means that there is a high cost incurred if 
the goal of the goal parameter (in tiiis case 100 Mbits/sec) is not mel. For a non-supply 
arc the resistance parameter for a given OD/6>o.S' combination is set lo the inverse of the 

25 arc capacity for the given OD/QoS combination. 

The two approaches to solve for flows for a given rouler module differ in 
how they deal with the goal parameters for non-supply arcs. The first approach solves 
for (low for one OD/OoS combination at a lime whereas the second approach solves for 
flow for all ()D/OoS combinations at once. For the first approach, the goal parameter 

30 for a non-supply arc is set to the negative of the historical flow for an OD/OoS 
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combination for ihc arc. The historical flow for an arc is the most recent flow amount 
recorded. Often times historical flows tend to change more slowly than demand or 
load, especially for arcs that have become a major artery for a data communication 
system. For instance, if the flow on a non-supply arc was most recently recorded as 50 
5 Mbits/sec for an OD/OoS combination, then the goal parameter for that arc would be 
-50Mbits/sec. 

With the first approach, for a given home router module 130, the 
dynamic dala flow splitter unit 220 determines historical flows from header information 
on data packets sent from the home router module. These historical flows of the home 

10 router module for sent packets are stored in the dynamic dala flow splitter unit 220. 
The historical flows are represented by averages, variances and other expressions 
kn(nvn in the art of statistics and mathematics. Hie neighborhood supervisor unit 214 
queries the dynamic data llow splitter unit 220 for the' historical flows of sent data 
packets as needed i'or flow calculations or to send to other neighboring router modules. 

1 ."^ The historical flows of sent dala packets are for arcs directly connected to the given 
home router module 130 handling packets sent by the home router module to networks 
or other router modules. In the depicted embodiment, the neighborhood supervisor unit 
214 of the given home router module 130 also receives updated historical flow 
information from other neighborhood supervisors for arcs that are directly connected to 

20 the given home router module and carry data packets to the given home router module. 
A special case of the first approach is where all historical flows are equal to zero. This 
special case occurs at such times as the initial starl-up phase of a data communication 
system. 

The second approach to solve for (lows for a given home router module 
25 130 uses computed flows instead of historical flows. Flows are determined for all 
OD/OoS combinations at once through an iterative process. For a A'^^' iteration, the 
overall How for an arc is determined by summing all flows for the OD/OoS 
combinations for the arc solved during the K^^^ iteration. The goal parameter for the 
non-supply arcs for the 1^^^ iteration are then equal to the negative of the overall 
30 flows for the arcs for the A.''^' iteration. 
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The techniques used to solve in this iterative fashion the flows for a 
home router module 130 lor all OD/OoS combinations at once require extensive 
compuling capability. Some embodiments use the second approach to solve for flows 
for all OD/OoS combinations at once. The depicted embodiment as incorporated into 

5 the methods of the described neighborhood supervisor unit 214 uses the first approach 
to solve Ibr flows one OD/OuS combination at a lime for a given home router module 
starling with Slep 710 of Figure 7. 

With the first approach, the neighborhood supervisor unit 214 of the 
home router module 130 collects potentials and historical flows from all neighboring 

10 router modules 130 direclly connected to the home router module in slep 720 of the 
flowchart of Figure 7 showing the method used by the depicted embodiment, fhe 
colk-cied poiemials and historical flows are for common OD/OnS combinations known 
lo both the home router module and at least one neighboring router module. Either the 
neighboring router mcidules or the home router module initiates transmission and 

15 receplinn of the potentials and historical flows collected from the neighboring router 
modules to the home router module. In the depicted embodiment, the neighboring 
rt»uter modules initiate the transmission of their potentials and historical flows to.lhe 
home router module. I iowever. in other embodiments, the home router module initiates 
the tninsmission. The dynamic load balancer unit 216 of the home router module 130 

20 then uses the collected poienlials and historical Hows along with the potentials and 
historical flows of the home router module to calculate, via the How respon.se function, 
the data packet flows for eacii OD/QoS combination to and from the home router 
module for all arcs connected to the home router module. For each connection between 
the home router module 130 and neighboring router modules or networks 120, there are 

25 two arcs as described above. One arc has the home router module as the origination 
node and the other arc as the destination node. In other embodiments, only one-sided 
connections are possible so that for each connection there is only one arc. 

The neighborhood supervisor unit 214 then u.ses the historical flows and 
demaiid or load flows to update the goal parameters for all the non-supply and supply 

30 arcs respectively of the home router module 130 in step 722 of Figure 7. Typically, the 
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goal parameters for non-supply arcs are set to the negative of the historical flow for a 
given OD/OoS combination and the goal parameters for the supply arcs arc set to the 
demand or load How on the arc. The neighborhood supervisor unit 214 also identifies 
any arcs that have zero overall flow based on their historical or demand/load flow so 
5 that any subsequent modifications of the home potentials arc done with the zero flow 
conditions in mind as explained further below. 

Initially, no OD/QoS combinations are selected by the neighborhood 
supervisor unit 214, so the outcome of decision 724 of Figure 7 regarding whether all 
OD/OoS combinations have been selected is ^^no." In such case, the neighborhood 

10 supervisor unil 214 selects an OD/OoS combination in step 726 that has not been 
chosen since the last time the start step 710 was performed. For all arcs cither 
originating from or terminating into the home router module, the neighborhood 
supervisor unil 214 of the home router module passes, in step 728, all the potentials of 
the home router module and those collected from neighboring router modules for the 

15 selected OD/OoS combination to the dynamic load balancer unit 216. The 
neighborhood supervisor unit 214 also passes all the goal parameters and resistance 
parameters lor liie arcs of the particular router module 130 based on the most recent 
historical and demand/load flows collected by the neighborhood supervisor unit 214 to 
the dynamic load balancer unit 216. 

20 The dynamic load balancer unit 216 then determines, for the selected 

OD/OoS combination, the Hows across the ares of the home router module. The 
dynamic load balancer unil 216 then determines, in decision step 730 of Figure 7 
whether the determined Hows are conserved for the selected OD/OoS combination or 
whether the time period for iteration has expired. Flows are conserved when the 

25 incoming flows plus flows originated by the home router module equals the flows 
leaving the home router module. If flows are not conserved for the selected OD/OoS 
combination or the time for iteration has not expired, the decision step in 730 is '^no^' 
and the method branches to step 732 where the dynamic load balancer unit 216 adjusts 
the potentials for the home router module 130 for the selected OD/OoS combination. 

30 After the home potentials are adjusted in step 732, flow conservation is again checked 
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in decision step 730. The home potentials are itcratively adjusted through the 
combination of potential adjustment in step 732 and flow conser\'ation verification in 
decision 730 until flow is conserved and decision step 730 is '^yes" and the method 
branches back to step 722 to update the goal parameters, iteration techniques to allow 
5 for rapid convergence to solutions involving iteration arc known in the art and include 
over or under-rclaxation techniques. In one embodiment, the potentials are adjusted 
one ai a lime by the iteration process for a given OD/OoS combination until How is 
conserved. In another embodimenl all the potentials for a router module for all the 
A A' combinations associated with the router module are adjusted for each iteration 

10 of a Ciaussian or other second order method of determining solutions by block 
lechnitiues knt>\vn in the art. 

If the neighborhood supervisor unit 214 has identified any zero flow arcs 
thai have zero overall flow when the goal parameters are inilially updated in step 722 
immedialely aller the historical and demand Hows are obtained in step 720. then during 

I 5 the home potential adju.siment in step 732. a goal parameter associated with a zero flow 
arc can be set to the negative of any calculated (low for the zero Hcnv arc resulting from 
adjustments of potentials for an OD/QoS combination to ensure that the flow for the 
zero flow arc remains small. The new value for the goal parameter for the zero flow arc 
is then used lo update the goal parameters in step 722. At the completion of adjusting 

20 potentials for all OD/OoS combinations, the updated goal parameter for a zero How arc 
will then be the summation of lltiws over all OD/OoS combinations for the particular 
zero flow arc due to adjustment of ptMentials. 

Once any goal parameters are updated in step 722 another OD/OoS 
combination is .selected which had not been previously selected since the last 

2.S performance of start step 710 and steps 728-732 are again performed for the newly 
selected OD/OoS combination. After steps 728-732 are completed, again any changed 
goal parameters are updated. The process similarly cycles through the OD/OoS 
combinations until decision step 724 determines that all OD/OoS combination have 
been selected, whereby the '^'es'■ branch is taken and decision step 734 determines if 

30 any potentials of the home router module 1 30 had been cidjusted by step 732. 
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If any home potentials were adjusted, then decision step 734 branches to 
step 736 of Figure 7 where the neighborhood supervisor unit 214 of the home router 
module 130 reports the adjusted home potentials and historical flows to the neighboring 
router modules. Decision step 738 then determines if the period of time since the 

5 neighborhood supervisor unit 214 of the home router module 130 last reported home 
potentials to neighboring router modules is greater than a predetermined amount of 
time, Tii. If the period is not greater then Tj^. decision step 738 branches back to the 
start step 710. If the period is greater than Tn. decision step 738 branches to decision 
step 740, the same step to which decision step 734 branches if no home potentials were 

1 0 adjusted. In decision step 740 a detemiination is made as to whether any adjusted home 
pincntials have changed more than a predetermined threshold Ty since the last time 
electronic tables of the dynamic load balancer unit 216 storing the home potentials have 
been updated with adjusted home potentials. If any changes in the adjusted home 
potentials are greater than 7?, decision step 740 branches to step 742 wherein the 

15 electronic tables arc updated with the latest values for the home potentials. The 
dynamic load balancer unit 216 subsequently updates the routing table unit 218. If 
there are no changes in the adjusted home potentials greater than 7'/, decision step 740 
branches to the start step 710, 

The present invention is not limited to the particular routines used by the 

20 depicted embodiment to report updated potentials to neighboring router modules 130 or 
in updating electronic tables storing home potentials. Other embodiments use other 
methods, intervals, or thresholds to determine when to report to neighboring router 
modules I 30 and when to update electronic tables storing home potentials. 

In an alteniative embodiment, the resistance parameter for each 

25 particular arc is a function of the Hows of each OD/OoS combination. For instance, for 
a particular OD/OoS combination, the resistance. t\ is as follows: 

4/' 

where / is the flow for an arc for an OD/OoS combination and c is the capacity or 
residual capacity of the arc. It follows that since the response function expresses the 
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dependency of flow for an OD/QoS combination for an arc, the resistance parameters 
arc also expressible in terms of potentials. 

From the foregoing it will be appreciated that, although specific 
embodiments of the invention have been described herein for purposes of illustration, 
various modifications may be made without deviating from the spirit and scope of the 
invention. Accordingly, the invention is not limited except as by the appended claims. 
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CLAIMS 



1 . In a data communicalion system having network traffic with randomly 
disirihulcd demands and flows, a network traffic director system comprising: 

router modules configured to direct data packets in tiie data communication 
system, each router module being a home router module comprising: 

a neighborhood supervisor configured to send home potentials of the 
liome router module to neighborhood supervisors of neighboring router modules and to 
receive neighbor potentials of the neighboring router modules from neighborhood supervisors 
ol* neighboring router modules; 

a dynamic load balancer configured to determine flows based on the 
home anJ neighbor potentials, to adjust the home potentials if first conditions including fiow 
conditions are not met, and to update routing tables if second conditions based on the 
adjusted home potentials are met: and 

a dynamic data fiow splitter configured to receive data packets from 
netuorks and router modules, to select a portion of the data communication system for each 
data packet received based on the updated routing tables wherein each received data packet is 
transmitted to the portion of the data communication system selected by the dynamic data 
fiow splitter for the received data packet. 

2. The network tralTic director system of claim I wherein each of the 
home potentials of each home router module is associated with a pair of nodes of the data 
communication system comprising an origination node and a destination node. 

3. 'fhe network traffic director system of claim 1 wherein each of the 
home potentials ofeach home router module is associated with a quality of service level. 

4. The network traffic director system of claim 1 wherein the 
neighborhood supervisor is configured to send the home potentials to neighborhood 
supervisors of neighboring router modules when the flow conditions are not met. 
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5. The network traffic direclor system of claim 1 wherein the flow 
conditions comprise conservation of flows. 

6. The network trafflc director system of claim 1 wherein the dynamic 
load balancer is configured to determine flows by using a response function associated with 
optimizing at least one of a penally function or a merit function for stochastic demands and 
flows oi'thc data ci)mmunication system, 

7. The network traffic director system of claim 1 wherein the dynamic 
data flow splitter is conflgured to select the portion of the data communication system for 
each data packet received further based on a Markov prolocoL 

8. The network traffic direclor system of claim 1 wherein the dynamic 
data flow splitter is configured to select the portion of the data communication system for 
each data packet received further based on a Routing Wheel protocol. 

9. The network traffic direclor system ol' claim 1 wherein the dynamic 
data How splitter is further configured to select a portion of the data communication system 
based on previously selected portions of the data communication system when an address for 
a received data packet is a same type as an address stored in the dynamic data flow splitter. 

10. The network traffic director syslem of claim I wherein the dynamic 
data flow splitler is further configured lo select of a portion of the data communication 
system based on a quality of service level associated with a data packet. 

11. In a communication system having network traffic with flows, a 
nelwork traffic direclor syslem comprising: 

router modules, each router module being a home router module configured to 
store and adjust home potentials of the home router module, and configured lo receive and 
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store neighbor potentials of neighboring router modules, the home router modules 
configured to determine ideal data flows using the home and neighbor potentials with an 
optimization of at least one of a merit function or a penalty function involving stochastic 
changes in at least one of demands or topology in the communication system, the home router 
modules configiired to receive and route network traffic based on the home and neighbor 
potentials. 

12. The network traffic director system of claim 11 wherein home router 
modules arc ct>n figured to be a neighbor router module of other home router modules. 

ITv The network traffic director system of claim 11 wherein the router 
modules arc c<Mifigurcd to continue adjusting home values to seek flow conservation until 
fiow conservation is satisfied or until a lime period has expired. 

14 The network traffic director system of claim 1 I wherein the merit or 
penali) function arc at least locally approximated by a quadratic function of the fiows. 

15. I he network traffic director system of claim 1 1 wherein the neighbor 
potentials arc stored according to an aggregation scheme. 

16. The network traffic director system of claim 1 1 wherein the ideal data 
fiows arc further ba.sed on resistance of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of arc capacity. 

17. The network traffic director system of claim 1 1 wherein the ideal data 
Hows are further based on resistance of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of Hows of the arc. 
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18. The network trarPic director system of claim 1 1 wherein the neighbor 
router modules arc connected to the home router modules without any other router modules 
therebetween. 

19. Tlie network traffic director system of claim 1 1 wherein the neighbor 
router modules are configured to be associated with the home router module through at least 
one of a political, organi/^alional, topological, topical, or geographical relation. 

20. The network traffic director system of claim 1 1 wherein each router 
module is configured to adju.sl home polenlials to an equilibrium point for a difference 
function associated uilh Hows of arcs associated with the router module, the dilTerence 
function as.sociated with differences between home potentials and neighbor potentials, 

21. flic network traffic director system of claim 11 wherein each router 
module stores adjusted home values when adjustment of home values is greater than a 
threshold. ^ 

22. I he network tralTic director system of claim I 1 wherein each home 
router module iransnuts adjusted home values to its neighbor router modules based on how 
much time has elapsed since last transmitting adjusted home values. 

23. 1 he network traffic director system oi' claim 11 wherein the merit 
fiinclion or penalty function involves costs of congestion comprising at least one ol trallic 
delays, high latency, diminished throughput, lost packets, or unresponsiveness to sudden 
changes in topology or loading. 

24. The network traffic director system of claim 11 wherein the neighbor 
router modules are connected to the home router modules wherein each connection has only 
one arc. 
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25. In a data communication system having network traffic, a network 
traffic director method comprising: 

storing home potentials associated with a home router module; 

receiving and storing neighbor potentials associated with neighbor router 
modules of the home router module; 

determining ideal flows based on the home and neighbor potentials and 
optimizing at least one of a merit function or a penalty function for random changes in 
demands or topology in the data communication system; 

determining conservation of flows for the home router module; 

adjusting the home potentials to approach a state where Hows are conserved ii' 
determined not to be conserved; 

reporting adjusted home potentials to the neighbor router modules; and 

routing network traffic based on home and neighbor values, 

26. The network traffic director method of claim 25 wherein the routing 
further includes using at least one of a Markov protocol, a Routing Wheel protocol, stored 
data packet addresses, or a level ol" quality of service of received communication traffic to be 
routed. 

27. The network traffic director method of claim 25 wherein the 
determining ideal Hows further involves a dilTerenee function associated with fiows. home 
potentials, and neighbor potentials. 

28. f he network traffic director method of claim 25 wherein the merit or 
penalty functions are at least locally approximated by a quadratic function of fiows. 



29. In a data communication system having network traffic, a network 
traffic director system comprising: 

means for storing home potentials associated with a home router module; 
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means for receiving and storing neighbor potentials associated with neighbor 
rouier modules of the home router module; 

means for determining ideal flows based on the home and neighbor potentials 
and optimizing at least one of a merit function or a penally function for random changes in 
demands or topology in the data communication system; 

means for determining conservation of flows for the home router module; 

means for adjusting the home potentials lo approach a slate where (lows are 
conserved if determined not lo be conserved: 

means for reporting adjusted home potentials to the neighbor router modules: 

and 

means lor routing network traffic based on home and neighbor values. 
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